Some rigorous results for the Hopfield Model 
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Abstract 

For the Hopfield model with the Hebb connection matrix we investigate the case of p memorized 
patterns that are distorted copies of the same standard. In other words, we try to simulate that 
learning always takes place by means of repeating presentations of one and the same standard, and 
the presentations are accompanied by distortions of the standard. We obtain some rigorous results 
relating to the dependence of the fixed points on external parameters of the problem. 
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By the Hopfield model of a neural network we mean a dynamic system of n interacting " agents" . The 
agents can take only two values ai = ±1, i = 1,2, ... ,n, and they are connected with each other by the 
Hebb connection matrix 

! J = S"^ ■ S, J,, = 0, i = 1,2,.. .,n. (1) 

The matrix S is (p x n)-matrix whose n-dimcnsional rows are the memorized patterns used for the network 
learning. 

Let us consider the following meaningful problem: a network had to be taught by p-time presentations 
of the standard, but errors crept into the learning process and, in fact, the network was taught by p 
distorted copies of the standard. What is the distortion's influence on the result of the learning? 
We formalize the problem choosing the matrix S in the form, 
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The n-dimensional rows of the matrix S, 



are treated as p distorted copies of the standard 
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The real number x is called the distortion parameter. 

Our goal is to find out how the set of fixed points depends on the distortions of the standard (because 
the set of fixed points represents the "memory" of the network). In other words, we investigate the ability 
of the Hopfield model to generalization in the specific case of one distorted standard. 

Only the first p coordinates of the standard are distorted, and the remained q — n — p coordinates 
are not distorted. The problem (l)-(4) is discussed in Section 2. Of course, the distortions are rather 
specific: for each memorized pattern only one coordinate of the standard is distorted and the value of the 
distortion x is the same for all the memorized patterns. In the next Sections we generalize this simple 
model. 

The asynchronous dynamics of the network is supposed. The state of the network as a whole is 
described by the configuration vector 

a = (cti, (T2, . . . , cr„). 
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2 Basic Model 



The problem (l)-(4) will be called the Basic Model. It was investigated in detail in [Q. The results of 
are summarized in items A)-C). 

A) . A necessary condition for a configuration vector to be a fixed point is that its last q coordinates 
be equal to each other. 

Consequently, fixed points can be chosen in the form 



(7* = (cti, (T2, . . . , o-p, 1, . . . , 1); 



(5) 



Here, a' = (ui, a2, . . . , Cp) is the p-dimensional part of the configuration vector a* . This p-dimensional 
vector will be used below. 

B). // a vector a* (5) is a fixed point, then every configuration vector of the form (5) whose distance 
from the standard is the same as the distance of a* is a fixed point. 

Consequently, we have to divide all the set of 2"^ vectors a* into classes E joining vectors equidistant 
from the standard. For the standard £*(n) these classes are 



(6) 
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The number of the classes Sfe is equal to p + 1. The number of the vectors in the class is equal to (^) . 
The configuration vectors from the same class are the fixed points simultaneously. 
C). Theorem 1: 

When X increases from —oo to oo, in consecutive order the set of the fixed points is exhausted by the 
classes 

Eq, El, ... , Efc^_^^ . 

The kth rebuilding from the class Efe_i to the class Efc occurs at the point x^: 



Xk^P- 



where 
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when £-4 < i; 

n— 1 3 ' 

when ^ > i 

n — 1 3 
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In Fig.l the graphical illustration of Theorem 1 is shown. 

The case p = n worth to be mentioned specially. Here all the rebuilding points Xk stick to one point. 



n + 1 



(9) 



For any x from the left side of ^ there is only one fixed point. It is the standard £{n) = Eq. For x from 
the right side of ^ every configuration vector from the class Er ^+i i is the fixed point; in other words, in 

^ [ 2 J 

this region there are (j:i±ij) fixed points. 

3 Complete Generalization 

1°. The standard £{n) can be replaced by an arbitrary configuration vector 

a = (ai, a2, . . . , Q!„), ai = {±1}. 
In this case the memorized patterns have the form 

s^'^ = (ai, . . . , (1 - x)ai, an), I = 1,2,..., p. 
All the statements of items A)-C) remain valid, but now the vectors a* (5) have the form 

(T* = (aiCTi, a2cr2, • • ■ , OipOp, ap+i, . . ., an). 



Thus, £{n) can be used as the standard without the loss of generahty. 

2° . Suppose the memorized patterns are obtained from the standard e{rL) by identical and simultaneous 
distortions of its m (but not only one!) coordinates, 
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Of course, now n — p-m + q. It is not difficult to show that the fixed points have the form of the piecewise 
constant vectors 

<?* = (o-i, ■ • ■ ,cri,(72, ■ ■ ■ , o-2 _, . . . , 0-p, . . . ,0-p , I, . . . , 1). 

m m q 

As above these vectors are grouped into classes S^™-* analogous to the classes Sfc (6). The only difference 
is that now the value —I has to be assigned not to a separate coordinate ct/, but to m coordinates of the 
same name. Again, the number of the vectors a* in the class S^™"* is equal to (^). Then we have the 
generalization of the Theorem 1: 

The value of the parameter x corresponding to the kth rebuilding from the class S^™^^^ to the class S^™'' is 

^-{2k-l) _ _ , f 

— P ' 2{2k ly' ^ ^ ^' ■ ■ ■ ' ^"ioa;, k„iax — min ^p, 

3°. The memorized patterns (3) can be rotated as a whole. Suppose the rotation matrix U — (uij) 
acts on the first p coordinates of n-dimensional vectors only. Then the standard takes the form 

U= (lil,U2,---,Wp, 1, ■■•,!), (10) 

u' 

where 

p 

ui^^uii, l^l,2,...,p. (11) 

i=l 

In Eq.(lO) the vector u' is the p-dimensional part of new standard u (the same as the vector a' in Eq.(5) 
is the p-dimensional part of the vector a*). It is easy to see that || u' |p= p. 
At the same time, the memorized patterns (3) take the form 

S*-'^ = (Ui - XUii,U2 - XU21, ...,Up- XUpi, 1, . . . , 1), I = 1,2, . . . ,p. (12) 

The elements of the relevant connection matrix J^^^^ are 

j{U) J , / Mi , when I <i <p 

= •JiiO.ia.i, where fli = < , , . , 

«J « J ' [ 1 , when p <i <n, 

and the matrix elements J.^ are determined by Eqs.(l),(2). 

Let us suppose that the standard £{n) remains unchanged after the rotation: 

ui = 1. 

Because the connection matrix J^'^-' coincides with the initial matrix J, all the statements of the items 
A)-C) of the Basic Model remain unchanged too. Note, here all the first p coordinates of the memorized 
patterns (12) are distorted. Consequently, we succeeded in getting rid of one of the hmitation of the 
Basic Model (see Introduction). 



4 Ground State 

It is well-known that an energy E = ^ij ' '^i ' '^^^ associated with every state S of 

a network. The energy decreases during the network evolution. The fixed points are the local minima 
of the energy E. A fixed point that is the global minimum of the energy is called the ground state of a 
network. 



It is not difficult to show that for the cases discussed above, all the states that are the fixed points 
simultaneously have the same energies. In other words, every time the set of fixed points is the ground 
state of the network, and there are no local extremums. This is due to the symmetry of the problems in 
question. When the symmetry of the problem is reduced, it is possible that different fixed points have 
different energies. Often just the ground state is of interest, because its energy is minimal. From my 
point of view namely the ground state has to be treated as the result of learning. Only the ground state 
is discussed in this Section. 

1°. Let us examine the problem of the item 3° from the previous Section for the case when the 
standard s{n) changes due to rotation: 

Again, as in the item A), only configuration vectors a* (5) can be the fixed points. Again, the statement 
of the item B) is valid, however now it is true not for all the fixed points, but for the ground state of the 
network only: 

// a vector a* (5) is a ground state, then every configuration vector of the form (5) whose distance from 
the standard u is the same as the distance of a* is a ground state. 

Consequently, we have to divide the set of the vectors a* into classes joining the vectors that are 
equidistant from the standard u. Only now the vectors that are equidistant from u are vectors a* with 
the same value of the cosine of the angle between p-dimensional vectors tr' and H' , 

cos^=ai^= (-;-^;) ds) 

P II II • II w II 

Let the vectors a* be grouped into classes E^^^ , 



4"' 



aj ■ Ui = p • coswk > , k = 0,1, . . . ,t, 



i=l 



where f + 1 is the number of the different values of the cosine (13). Without loss of generality we can 
assume that the cosines are arranged in decreasing order, 

cos Wo > coswi > . . . > COSWt- 

Then it is easy to see, that cos Wk = — cos Wt-k, Vfc < t. Therefore, the cosines are negative beginning 

from some number k. Finally, the generalization of the Theorem 1 from the item C) is: 

When X increases from — oo to oo, in consecutive order the ground state is exhausted by the classes 

yiU) yiU) y{U) 

The kth rebuilding of the ground state from the class S^^-*^ to the class S^^^ occurs at the point Xk, 

fc = 1, 2, . . . , kmax- (l'^) 



P 

Xk = ^ 



1+ ' 



p ■ (cos Wk-1 + cos Wk) 



If Xi > jp, then kraax = t- If xx < jp, the rebuildings come to an end when the denominator in Eq.(14) 
becomes negative (kmax <t). 

Note. When the distortion x belongs to the interval {xk,Xk+i), all the configuration vectors from the 
class S^^^ are the ground state of the network. But the compositions of the classes S^^^ are determined 
by the values of {wjjf^i (11) only. And the choice of {ui}f^i is in the researcher's hand. In other words, 
selecting and the distortion parameter x, the network with a preassigned ground state can be 

created. 

For example, let us create a network with the ground state 

d={ -1, -1, 1, 1, 1, 1 ) 

/3 = ( 1, 1,-1, 1, 1, 1 ) 

7=( 1, 1, 1, -1, 1, 1 )• 



At first wc need to find the vector u of the form (fO) equidistant from the vectors a, /? and 7 (and only 
from these vectors!). This problem has a lot of solutions. For example, we can take the vector u in the 
form of 

p 

u = {u, u, 2m, 2u, U5, . . . , Up, 1, . . . , 1), where lOw^ + ^ uf = p. 

1=5 

The minimal p satisfying this equation is equal to 4. Consequently, we have 



u = (u, u, 2u, 2ii, 1, . . . , 1), u 



Next, 2^ configuration vectors a* (5) split into classes which join the vectors equidistant from the 

found standard it. One of these classes consists from our vectors a, (3 and 7 exactly. It is not difficult 
to determine the number of this class. Indeed, the first four values of the cosines (13) and the relevant 
classes E^^^ are: 

cos W{) = 6u/p = 3u/2 



( 1, 1, 1, 1, !,...,!) eR4+q 



cos Wi = COS W() — 2ii/p = u 
(c/) _ / ( -1, 1, 1, 1, 1,...,1) 

( 1, -1, 1, 1, 1,...,1) 



C0SW2 = cos Wo — 4:u/p = u/2 
( -1, -1, 1, 

sr' = <! ( 1, 1, -1, 



( 



1, 



1, 1,, 
1, 1,, 
-1, 1,. 



.1) 
.1) 
,1) 



C0SW3 



cos Wo ^ 6u/p = 

' ( -1, 1, -1, 1, 1,...,1) 

( -1, 1, 1, -1, 1,...,1) 

( 1, -1, -1, 1, 1,...,1) 

( 1, -1, 1, -1, 1,...,1) 



Thus, the class ^2'^'' contains the given configuration vectors a, f3 and 7 only. If we take four memorized 
patterns of the form (12) and the distortion parameter x inside the interval (0:2, 0:3), the relevant network 
has the preassigned ground state. 

To define the memorized patterns s^'^ we need the matrix elements uu of the rotation matrix U. 
This matrix transforms the vector s{n) into the standard u. More exactly, it is necessary to know only 
the upper (4 x 4)-block of this matrix realizing the nontrivial part of the rotation. In other words, it is 
necessary to solve the problem 
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U(4x4) 



This problem has a lot of solutions. For example, 

U(4x4) ^ 



where u 



\ 



-u/2 


3u/2 




3zt/2 
3u/2 

u/2 
u/2 



\ 

-u/2 



3u/2 / 



Consequently, as the memorized patterns (12) (4 + g)-dimensional vectors 



S<2) 
S<3) 



( u{l + x/2), u, u{2-3x/2), 

( u, u{l-3x/2), 2u, 

( u(l-3a;/2), u, u{2-x/2), 

( u, u(l + a;/2), 2u, 



2m, 
m(2 - x/2), 
2u, 

u{2-3x/2), 



1, . 
1,. 
1,. 
1,. 



,1) 
.,1), 



can be used. If x is from the interval 2 (1 + g/(g + 6w)) < a; < 2 (1 + (//(q + 2m)), the configuration 
vectors a, (3 and 7 are the ground state of the Hopfield network with the connection matrix (1). The 
minimal n for such a network is equal to 5, n = 5. This corresponds to g = 1. If g = 0, we have the 
equality p = n, and, consequently, all the rebuilding points stick to the point ^ (see the comment to 
Eq.(9)). 

2°. Suppose the asynchronous dynamics is defined as 

cri(i + 1) = sgn ( - • 51 Ji3<^jit) + H 

where the connection matrix J is given by Eqs.(l),(2). The parameter H is called the dynamic threshold 
HJ. Usually, in theoretical considerations it is used to eliminate the linear term in the energy E (such 
a term appears when we turn from the (0, l)-network to the (—1, +l)-network). Apparently, the role of 
the dynamic threshold is much more important. In the case of 7^ is analyzed in details. For the 
sake of simplicity, here I present the results obtained for iJ > only. 
Theorem 2. 

Let X € {xk,Xk+i). When H increases from its initial zero value to infinity, in consecutive order the 
ground state of a network is exhausted by the classes 

The rebuilding of the ground state from the class Yik+i-i to the class occurs when H is equal to 

rr , , n+p-2(2(fc+l-.)-l) ^ ■ 1 9 . 

n 

Here, the classes Sfe are defined by Eq.(6) and the rebuilding points x^ are given by Eq.(7). In Fig. 2 
for some values of p and n the phase diagram for the ground state is shown. We see that changing H 
purposefully, we can change the ground state significantly. In particular, the standard ein) = Sq can be 
done the ground state of the network. It is important to understand either this result depends on the 
specific form of our connection matrix, or it is more general. In other words, is it possible to choose the 
thresholds in the general Hopfield model in such a way that memorized patterns will necessarily be its 
fixed points? The positive answer seems to be very probable. 

3° . I generalize the Basic Model for the case of different distortions of all the memorized patterns : 

s<') = (1,..., 1,1-^, !,...,!), Z = l,2,...,p. 
I 

Suppose, that 



X 



> > . . . > > > > . . . > x'^^\ 



and p+ is the number of positive distortions a;''^ . Theorem 3 is an analog of the statements A)-C) from 
the Basic Model. 
Theorem 3. 



a) Only configuration vectors 

a*(fc) (-1,-1,..., -1,1,...,!), 0,1,..., kma.: (15) 



can be the ground .state, where 



kmax = min p+ 



n+p + 2' 



4 

b) For the vector S*{k) to be a ground state, it is necessary and sufficient to have 

k p k—l p 

n + p-2(2fc+l) ^ n + p-2(2fc-l) ■ ^ ^ 



The inequalities (16) are very useful. For example, suppose p <^ n, and n is so large that in Eq.(16) 
the terms containing n in the denominators can be omitted. Then, the statement b) is simplified: 

S* [k) is the ground state if and only if x^'^^^^ < p < a;*-'"-'. (17) 

In particular, 

f7*(0) is the ground state if and only if x^^'^ < p. (18) 

These statements are very interesting. Namely, Eqs.(17) and (18) connect the quality of learning with 
the relation between the number of presentations p of the standard and the values of distortions. (The 
distortions are measured in bits.) In particular, Eq.(18) means that the network understands the standard 
e{n) = (T*(0) correctly, if and only if the number of its presentations p exceeds the maximum distortion 
x^^\ It seems, this result clarifies qualitatively why the well-known Latin saying "Repetitio est mater 
studiorum" is true. Indeed, to be sure that a network with the Hebb connection matrix understands a 
standard, we have to show it repeatedly. Also, we must be sure that the number of presentations p is 
greater than the maximal distortion x^^\ Since usually the distortions are Gaussianly distributed, the 
last requirement can be realized always. 

Next, we can rewrite the inequalities (16) in the form 

Xk < x^"^ - dW; xC'+i) - dC^+i) < Xk+i, (19) 

where 



+ p- 2(2fc - 1) 

Sometimes Eqs.(19) are more suitable, since they relate the distortions a;''''-' with the rebuilding points 
Xk from Eq.(7). When the distortions are known, with the aid of Eqs.(19) the ground state can be easily 
found graphically. 

Let us clarify the last statement with the aid of a simple example. Let p = 5 and n is so large that 
kmax = 5 and d'*^-' oc o(l/n). Only one of the configuration vectors (J*{k), fc 0, 1, . . . , 5 of the form (15) 
can be the ground state. We make use of Eq.(19). We write down the necessary and sufficient conditions 
for each a*{k) to be a ground state (here d'^'^^ can be omitted): 



(T*(0) ^ a;(i)<a; 



CT*(1) <=^ XI <x^^^ 

a* (2) ^ a;2<a;(2) 

a* (3) ^ X3< 

CT*(4) ^ X4< 

(T*(5) X5 < x'^^\ 



< X2- 

< X4. 
x'-^^ < X5. 



Next, let the rebuilding points Xk and the distortions x^'^'^ be located as it is shown in Fig. 3. Since only 
for k = 2 the inequalities (19) are fulfilled, the vector (j*{2) = (—1, —1, 1, . . . , 1) is the ground state. 



5 Summary 

We present rigorous results relating to the model (l)-(4) and its generalizations. Computer simulations 
confirm these results. The proofs of the statements of Sections 2-4 are in j^, 1^. A meaningful 
interpretation of the obtained results can be found in the same references. In the present publication we 
do not include this topic deliberately with the exception of "the proof of the Latin saying. 
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Fig.l. The rebuilding points Xk from Eq.(7) divide the axis x into intervals inside which the set of the 
fixed points belongs to different classes (see Theorem 1). 

Fig. 2. The straight lines Hi{x) from Theorem 2 divide the half-plane {x,H > 0) into regions inside 
which the ground state belongs to different classes S^; p = 3, n = 9. 

Fig. 3. The example of the location of the rebuilding points Xk from Eq.(7) and the different distortions 
2;(*:) from Theorem 3. 
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